Repository navigation
feat(build_stream): complete ER-BSM-003 log observability implementation - #5809
Open
SOWJANYAJAGADISH123 wants to merge 15 commits into
Open
SOWJANYAJAGADISH123 wants to merge 15 commits into
SOWJANYAJAGADISH123 wants to merge 15 commits into
Conversation
…ell#5803) * feat(build_stream): add log query API with response-time redaction Implements a read-only log query API for Build Stream that allows authenticated clients to retrieve stage log content with response-time redaction of sensitive data. The API enforces job ownership, path traversal protection, rate limiting, and concurrency limits. Existing job detail endpoint is extended with log metadata fields. Related: dell#5733 * revert(test): remove log_query FVT from this PR FVT code has not been validated in a lab environment yet and will be included in a follow-up PR.
SOWJANYAJAGADISH123
requested review from
Rajeshkumar-s2,
RvishankarOMnia,
VenkateswaraVatam,
abhishek-sa1 and
snarthan
as code owners
November 15, 2026 03:19
…03-log-query-api-and-redaction) Implement FR-3.1 write-time credential redaction. execute_playbook no longer sets ANSIBLE_LOG_PATH; it captures merged stdout/stderr via subprocess.Popen, redacts Tier 1 secrets line-by-line through the shared RedactionService, and writes only sanitized output to disk. Handles non-UTF-8 (errors=replace), multi-line PEM key blocks (streaming state), timeout drain, and fail-closed line suppression. Applies the same redaction to the validate stage output. Adds RedactionService.redact_stream_line + StreamRedactionState and unit tests for the streaming redactor and the watcher capture path. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…M-003-ci-console-and-parity)
Add a self-contained default.after_script to the build, cadence, and deploy
child CI templates (AC-006/AC-007). On success and failure it previews the last
N execution-log lines (BSM_LOG_TAIL_LINES, default 50) for stages that ran an
Ansible playbook via GET /jobs/{id}/logs, and prints authenticated curl command
templates with a <TOKEN> placeholder (no credentials echoed). Runs in its own
shell, reads the freshest token from /tmp or the ACCESS_TOKEN dotenv var, maps
GitLab stage -> BSM stage per pipeline, and never changes the job exit status.
Self-hosted and external modes share these templates (AC-008 parity).
Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…SM-003-ci-console-and-parity) Runnable verification of the default.after_script embedded in the build, cadence, and deploy-child CI templates (AC-006/AC-007/AC-008). Drives the real POSIX shell against a mock BSM log API: asserts the labelled tail preview, custom BSM_LOG_TAIL_LINES, <TOKEN> CLI templates with no credential echo, non-fatal API-unreachable handling, 404 graceful message, non-playbook-stage skip, and cross-template parity. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…-query-api-and-redaction) Document the [log_query] configuration section in build_stream.ini and the BSM_LOG_TAIL_LINES CI variable in INPUT_CONTRACT.md §1.3. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
Silence pylint import-position warnings and line-length findings in the new test modules. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…ig.yml (ER-BSM-003-log-query-api-and-redaction) Expose log_tail_lines_default, log_tail_lines_max, max_log_response_bytes, log_query_rate_limit and log_query_max_concurrent as operator inputs in build_stream_config.yml (Section 5), with JSON schema, L2 validation and INPUT_CONTRACT docs. The log query API reads them at startup and falls back to build_stream.ini/defaults on invalid values. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…pt/omnia (ER-BSM-003-log-query-api-and-redaction) Add tests for the log inputs (schema, validator, overlay, precedence). Isolate test_upload_use_case.py destinations in tmp_path; it previously overwrote the real orchestrator network_spec.yml and pxe_mapping_file.csv with test content. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…SM-003-ci-console-and-parity) Add log_ci_preview_lines to build_stream_config.yml with schema and L2 validation (must not exceed log_tail_lines_max). The hosted_gitlab role pushes it to the GitLab project as BSM_LOG_TAIL_LINES on every run, and the CI templates no longer hardcode the value. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…ages (ER-BSM-003-log-query-api-and-redaction) Unfiltered log queries list only create-local-repository, build-image, validate, restart and deploy. Filtering on parse-catalog or upload returns 404 explaining the stage has no execution log, and job detail reports has_log=false for them. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…ER-BSM-003-log-query-api-and-redaction) Cover the playbook-stage-only rule and automate TC-NFT-001/002/003 latency budgets in-process. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…SM-003-ci-console-and-parity) Access tokens expire after 60 minutes, so after long playbook stages the after_script preview got 401. Mint a fresh token from CLIENT_ID/CLIENT_SECRET when the saved token is missing or rejected, in the build, cadence and deploy-child templates. Credentials are never printed and the hook never changes the job status. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
…R-BSM-003-ci-console-and-parity) Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
SOWJANYAJAGADISH123
force-pushed
the
release_2.3.0.0-rc2.1
branch
3 times, most recently
from
November 15, 2026 03:46
4025f52 to
3e8d136
Compare
…ter_script (ER-BSM-003) - publish_stage_log: use copyfile + chmod 0644 + chcon instead of copy2 so the job log copy is not left as var_log_t/0600 (unreadable by the BSM container, which made the logs API return 404 / has_log=false). - CI after_script: fall back to BSM_CLIENT_ID/BSM_CLIENT_SECRET when CLIENT_ID/CLIENT_SECRET are unset so a token can be minted. - deploy child pipeline: export CLIENT_ID/CLIENT_SECRET from BSM_CLIENT_* so deploy/restart/validate jobs can fetch the execution log preview. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
The deploy_openchami role resolved the latest release tag via the GitHub API and built <tag>/openchami-0.2.0.rpm. When the latest tag moved to v0.2.1 the asset name stayed openchami-0.2.0.rpm, so the URL returned HTTP 404 and OpenCHAMI installation failed after all retries. Use a fixed openchami_rpm_url (v0.2.0) and drop the release-metadata lookup and the URL-derivation task. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the Solution
Implement the remaining ER-BSM-003 log observability work: write-time credential redaction in the playbook watcher, CI console log preview with CLI templates, and comprehensive test automation. This ensures logs are redacted before disk writes and provides operators with actionable log previews in GitLab pipelines.
Related Issue
Changes
Core Logs (
src/build_stream/app/core/logs/redaction.py)StreamRedactionStateandredact_stream_line()for line-by-line streaming redactionPlaybook Watcher (
src/build_stream/app/playbook-watcher/playbook_watcher_service.py)ANSIBLE_LOG_PATHfrom Ansible subprocess environmentsubprocess.Popenwith reader threadCI Templates (
src/build_stream/roles/hosted_gitlab/files/)default.after_scriptto.gitlab-ci-build.yml,.gitlab-ci-cadence.yml,.gitlab-ci-deploy-child-template.ymlBSM_LOG_TAIL_LINES, default 50) for Ansible-backed stagescurltemplates with<TOKEN>placeholder (no credential echo)Test Automation
test/build_stream/nft/unit/playbook_watcher/test_write_time_redaction.py(8 real-subprocess tests)test/build_stream/nft/unit/ci_console/test_after_script_log_preview.py(10 mock-server tests)test/build_stream/nft/unit/core/logs/test_redaction.py(streaming redactor tests)test/build_stream/nft/unit/playbook_watcher/test_stage_log_paths.py(Popen harness)Documentation (
src/build_stream/docs/contracts/INPUT_CONTRACT.md)[log_query]configuration section inbuild_stream.iniBSM_LOG_TAIL_LINESCI variableFiles Changed
src/build_stream/app/core/logs/redaction.pysrc/build_stream/app/playbook-watcher/playbook_watcher_service.pysrc/build_stream/docs/contracts/INPUT_CONTRACT.mdsrc/build_stream/roles/hosted_gitlab/files/.gitlab-ci-build.ymlsrc/build_stream/roles/hosted_gitlab/files/.gitlab-ci-cadence.ymlsrc/build_stream/roles/hosted_gitlab/files/.gitlab-ci-deploy-child-template.ymltest/build_stream/nft/unit/ci_console/__init__.pytest/build_stream/nft/unit/ci_console/test_after_script_log_preview.pytest/build_stream/nft/unit/core/logs/test_redaction.pytest/build_stream/nft/unit/playbook_watcher/test_stage_log_paths.pytest/build_stream/nft/unit/playbook_watcher/test_write_time_redaction.pyTesting
sh -nvalidated on allafter_scriptblocksBackward Compatibility
BSM_LOG_TAIL_LINESdefaults to 50 if not setSuggested Reviewers
@build_stream-team@security-reviewer@ci-team